Building Middleware for Real-Time Dependable Distributed Services

We consider a real-time, distributed service to be dependable if it continues to have timely, predictable behavior even in the presence of partial failures. Services with this property are desirable in a host of real-time scenarios, including factory floor automation, medical monitoring equipment, and combat systems. Most distributed services built with contemporary fault-tolerance toolkits are not dependable; they exhibit unpredictable, albeit logically correct, behavioral patterns under failure conditions. We have designed and implemented middleware explicitly for real-time dependable services. We aimed at maintaining sub-second worst-case guarantees for failure detection and recovery, even when failures conspire with network load and CPU load to undermine determinism. The paper reports our experience in marrying software fault tolerance and real-time disciplines, from the definition of the requirements to the characterization of the resulting system.